Solved: Cisco 6509 Total output drops

stanleylin1986 · ‎09-15-2015

Hi All

I have noticed that a number of the interfaces on a Cisco 6509 switch are reporting output drops. The links to connect to Cisco 3524 on the other floods.

The interface config is:

6509-OA#sh run int gi 7/46
interface GigabitEthernet7/46
description
switchport
switchport mode trunk
no ip address
end

6509-OA#sh int gi 7/46
GigabitEthernet7/46 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 0011.9298.d3a1 (bia 0011.9298.d3a1)
Description: A-HQ14F-A020620-OP
MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
reliability 255/255, txload 64/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:33, output 00:00:02, output hang never
Last clearing of "show interface" counters 5w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 236672777
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 68000 bits/sec, 34 packets/sec
5 minute output rate 25429000 bits/sec, 5312 packets/sec
173333928 packets input, 58845989538 bytes, 0 no buffer
Received 55386189 broadcasts (49023862 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
5921612450 packets output, 3730501839036 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

6509-OA#sh queueing int gi 7/46
Interface GigabitEthernet7/46 queueing strategy: Weighted Round-Robin

QoS is disabled globally
Port is untrusted
Extend trust state: not trusted [COS = 0]
Default COS is 0
Queueing Mode In Tx direction: mode-cos
Transmit queues [type = 1p2q2t]:
Queue Id Scheduling Num of thresholds
-----------------------------------------
1 WRR low 2
2 WRR high 2
3 Priority 1

WRR bandwidth ratios: 255[queue 1] 1[queue 2]
queue-limit ratios: 100[queue 1] 0[queue 2] 0[Pri Queue]*same as Q2

queue random-detect-min-thresholds
----------------------------------
1 100[1] 100[2]
2 100[1] 100[2]

queue random-detect-max-thresholds
----------------------------------
1 100[1] 100[2]
2 100[1] 100[2]

queue thresh cos-map
---------------------------------------
1 1 0 1 2 3 4 5 6 7
1 2
2 1
2 2
3 1

Queueing Mode In Rx direction: mode-cos
Receive queues [type = 1q2t]:
Queue Id Scheduling Num of thresholds
-----------------------------------------
1 Standard 2

queue tail-drop-thresholds
--------------------------
1 100[1] 100[2]

queue thresh cos-map
---------------------------------------
1 1 0 1 2 3 4 5 6 7
1 2

Packets dropped on Transmit:
BPDU packets: 0

queue thresh dropped [cos-map]
---------------------------------------------------
1 1 33844276 [0 1 2 3 4 5 6 7 ]
* - shared transmit counter

Packets dropped on Receive:
BPDU packets: 0

queue thresh dropped [cos-map]
---------------------------------------------------
1 1 0 [0 1 2 3 4 5 6 7 ]

The QOS is disable, those drops are on transmit,

Any help much appreciated,

Many thanks

Stanley

InayathUlla Sharieff · ‎09-15-2015

Total output drops - The number of packets dropped because the output queue is full. A common cause of this might be traffic from a high bandwidth link being switched to a lower bandwidth link or traffic from multiple inbound links being switched to a single outbound link.
For example, if a large amount of bursty traffic comes in on a gigabit interface and is switched out to a 100Mbps interface, this might cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the incoming and outgoing bandwidths.

We need to make sure that the interfaces involved have the same settings
(duplex and speed). Also, another possible reason for this behavior is the
bursty traffic.

In this case, I would appreciate if you can send me the following command to
see if the counters have been incrementing:

-Switch#show count inter <interface number> delta
-Switch#show counter interface <interface number>

If necessary, it will be very helpful to get a packet capture from one of
the affected interfaces to see if bursty traffic is causing this.

Looking forward to hearing from you.

HTH

Regards

Inayath

View solution in original post

InayathUlla Sharieff · ‎09-15-2015

Total output drops - The number of packets dropped because the output queue is full. A common cause of this might be traffic from a high bandwidth link being switched to a lower bandwidth link or traffic from multiple inbound links being switched to a single outbound link.
For example, if a large amount of bursty traffic comes in on a gigabit interface and is switched out to a 100Mbps interface, this might cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the incoming and outgoing bandwidths.

We need to make sure that the interfaces involved have the same settings
(duplex and speed). Also, another possible reason for this behavior is the
bursty traffic.

In this case, I would appreciate if you can send me the following command to
see if the counters have been incrementing:

-Switch#show count inter <interface number> delta
-Switch#show counter interface <interface number>

If necessary, it will be very helpful to get a packet capture from one of
the affected interfaces to see if bursty traffic is causing this.

Looking forward to hearing from you.

HTH

Regards

Inayath

stanleylin1986 · ‎09-16-2015

Hi Inayath

Thanks for your reply,

6509-OA#sh count int gi 7/46 de
Time since last clear
---------------------
5w4d

64 bit counters:
0. rxHCTotalPkts = 174090142
1. txHCTotalPkts = 5982477845
2. rxHCUnicastPkts = 118432462
3. txHCUnicastPkts = 4801646331
4. rxHCMulticastPkts = 49265755
5. txHCMulticastPkts = 876429298
6. rxHCBroadcastPkts = 6391925
7. txHCBroadcastPkts = 304402216
8. rxHCOctets = 59026295492
9. txHCOctets = 3764703342842
10. rxTxHCPkts64Octets = 62104264
11. rxTxHCPkts65to127Octets = 2954833563
12. rxTxHCPkts128to255Octets = 315319623
13. rxTxHCPkts256to511Octets = 401723272
14. rxTxHCpkts512to1023Octets = 306937529
15. rxTxHCpkts1024to1518Octets = 2115649735
16. txHCTrunkFrames = 0
17. rxHCTrunkFrames = 174026407
18. rxHCDropEvents = 0

32 bit counters:
0. rxCRCAlignErrors = 0
1. rxUndersizedPkts = 0
2. rxOversizedPkts = 0
3. rxFragmentPkts = 0
4. rxJabbers = 0
5. txCollisions = 0
6. ifInErrors = 0
7. ifOutErrors = 0
8. ifInDiscards = 0
9. ifInUnknownProtos = 0
10. ifOutDiscards = 35870
11. txDelayExceededDiscards = 0
12. txCRC = 0
13. linkChange = 0
14. wrongEncapFrames = 0
All Port Counters
1. InPackets = 174090105
2. InOctets = 59026292189
3. InUcastPkts = 118432460
4. InMcastPkts = 49265724
5. InBcastPkts = 6391921
6. OutPackets = 5982468493
7. OutOctets = 3764697423498
8. OutUcastPkts = 4801638130
9. OutMcastPkts = 876428546
10. OutBcastPkts = 304401817
11. AlignErr = 0
12. FCSErr = 0
13. XmitErr = 0
14. RcvErr = 0
15. UnderSize = 0
16. SingleCol = 0
17. MultiCol = 0
18. LateCol = 0
19. ExcessiveCol = 0
20. CarrierSense = 0
21. Runts = 0
22. Giants = 0
23. InDiscards = 0
24. OutDiscards = 35870
25. InErrors = 0
26. OutErrors = 0
27. InUnknownProtos = 0
28. txCRC = 0
29. TrunkFramesTx = 0
30. TrunkFramesRx = 174026370
31. WrongEncap = 0
32. Broadcast_suppression_discards = 0
33. Multicast_suppression_discards = 0
34. Unicast_suppression_discards = 0
35. rxTxHCPkts64Octets = 62104228
36. rxTxHCPkts65to127Octets = 2954829055
37. rxTxHCPkts128to255Octets = 315319172
38. rxTxHCPkts256to511Octets = 401722675
39. rxTxHCpkts512to1023Octets = 306937019
40. rxTxHCpkts1024to1518Octets = 2115646450
41. DropEvents = 0
42. CRCAlignErrors = 0
43. UndersizedPkts = 0
44. OversizedPkts = 0
45. FragmentPkts = 0
46. Jabbers = 0
47. Collisions = 0
48. DelayExceededDiscards = 0
49. bpduOutlost = 0
50. qos0Outlost = 0
51. qos1Outlost = 0
52. qos2Outlost = 0
53. qos3Outlost = 34078406
54. qos4Outlost = 0
55. qos5Outlost = 0
56. qos6Outlost = 0
57. qos7Outlost = 0
58. qos8Outlost = 0
59. qos9Outlost = 0
60. qos10Outlost = 0
61. qos11Outlost = 0
62. qos12Outlost = 0
63. qos13Outlost = 0
64. qos14Outlost = 0
65. qos15Outlost = 0
66. qos16Outlost = 0
67. qos17Outlost = 0
68. qos18Outlost = 0
69. qos19Outlost = 0
70. qos20Outlost = 0
71. qos21Outlost = 0
72. qos22Outlost = 0
73. qos23Outlost = 0
74. qos24Outlost = 0
75. qos25Outlost = 0
76. qos26Outlost = 0
77. qos27Outlost = 0
78. qos28Outlost = 0
79. qos29Outlost = 0
80. qos30Outlost = 0
81. qos31Outlost = 0
82. bpduCbicOutlost = 0
83. qos0CbicOutlost = 0
84. qos1CbicOutlost = 0
85. qos2CbicOutlost = 0
86. qos3CbicOutlost = 0
87. bpduInlost = 0
88. qos0Inlost = 0
89. qos1Inlost = 0
90. qos2Inlost = 0
91. qos3Inlost = 0
92. qos4Inlost = 0
93. qos5Inlost = 0
94. qos6Inlost = 0
95. qos7Inlost = 0
96. qos8Inlost = 0
97. qos9Inlost = 0
98. qos10Inlost = 0
99. qos11Inlost = 0
100. qos12Inlost = 0
101. qos13Inlost = 0
102. qos14Inlost = 0
103. qos15Inlost = 0
104. pqueInlost = 0
105. Overruns = 0
106. maxIndex = 0
0. ifOutDiscards (ports 41-48) = 1484971157

6509-OA#sh count int gi 7/46
64 bit counters:
0. rxHCTotalPkts = 3771759251
1. txHCTotalPkts = 146925913040
2. rxHCUnicastPkts = 2822864782
3. txHCUnicastPkts = 118911997995
4. rxHCMulticastPkts = 825155443
5. txHCMulticastPkts = 20683966991
6. rxHCBroadcastPkts = 123739026
7. txHCBroadcastPkts = 7329948054
8. rxHCOctets = 1108686980959
9. txHCOctets = 94700702761568
10. rxTxHCPkts64Octets = 4963213362
11. rxTxHCPkts65to127Octets = 65889075379
12. rxTxHCPkts128to255Octets = 7461296882
13. rxTxHCPkts256to511Octets = 11164322578
14. rxTxHCpkts512to1023Octets = 8297389952
15. rxTxHCpkts1024to1518Octets = 52922374141
16. txHCTrunkFrames = 0
17. rxHCTrunkFrames = 3761189152
18. rxHCDropEvents = 0

32 bit counters:
0. rxCRCAlignErrors = 0
1. rxUndersizedPkts = 0
2. rxOversizedPkts = 0
3. rxFragmentPkts = 0
4. rxJabbers = 0
5. txCollisions = 0
6. ifInErrors = 0
7. ifOutErrors = 0
8. ifInDiscards = 0
9. ifInUnknownProtos = 0
10. ifOutDiscards = 2032422
11. txDelayExceededDiscards = 0
12. txCRC = 0
13. linkChange = 3
14. wrongEncapFrames = 0
All Port Counters
1. InPackets = 3771759216
2. InOctets = 1108686978367
3. InUcastPkts = 2822864779
4. InMcastPkts = 825155414
5. InBcastPkts = 123739023
6. OutPackets = 146925906587
7. OutOctets = 94700699188780
8. OutUcastPkts = 118911992419
9. OutMcastPkts = 20683966461
10. OutBcastPkts = 7329947707
11. AlignErr = 0
12. FCSErr = 0
13. XmitErr = 0
14. RcvErr = 0
15. UnderSize = 0
16. SingleCol = 0
17. MultiCol = 0
18. LateCol = 0
19. ExcessiveCol = 0
20. CarrierSense = 0
21. Runts = 0
22. Giants = 0
23. InDiscards = 0
24. OutDiscards = 2032422
25. InErrors = 0
26. OutErrors = 0
27. InUnknownProtos = 0
28. txCRC = 0
29. TrunkFramesTx = 0
30. TrunkFramesRx = 3761189117
31. WrongEncap = 0
32. Broadcast_suppression_discards = 0
33. Multicast_suppression_discards = 0
34. Unicast_suppression_discards = 0
35. rxTxHCPkts64Octets = 4963213337
36. rxTxHCPkts65to127Octets = 65889072036
37. rxTxHCPkts128to255Octets = 7461296593
38. rxTxHCPkts256to511Octets = 11164322030
39. rxTxHCpkts512to1023Octets = 8297389532
40. rxTxHCpkts1024to1518Octets = 52922372276
41. DropEvents = 0
42. CRCAlignErrors = 0
43. UndersizedPkts = 0
44. OversizedPkts = 0
45. FragmentPkts = 0
46. Jabbers = 0
47. Collisions = 0
48. DelayExceededDiscards = 0
49. bpduOutlost = 0
50. qos0Outlost = 0
51. qos1Outlost = 0
52. qos2Outlost = 0
53. qos3Outlost = 1396501965
54. qos4Outlost = 0
55. qos5Outlost = 0
56. qos6Outlost = 0
57. qos7Outlost = 0
58. qos8Outlost = 0
59. qos9Outlost = 0
60. qos10Outlost = 0
61. qos11Outlost = 0
62. qos12Outlost = 0
63. qos13Outlost = 0
64. qos14Outlost = 0
65. qos15Outlost = 0
66. qos16Outlost = 0
67. qos17Outlost = 0
68. qos18Outlost = 0
69. qos19Outlost = 0
70. qos20Outlost = 0
71. qos21Outlost = 0
72. qos22Outlost = 0
73. qos23Outlost = 0
74. qos24Outlost = 0
75. qos25Outlost = 0
76. qos26Outlost = 0
77. qos27Outlost = 0
78. qos28Outlost = 0
79. qos29Outlost = 0
80. qos30Outlost = 0
81. qos31Outlost = 0
82. bpduCbicOutlost = 0
83. qos0CbicOutlost = 0
84. qos1CbicOutlost = 314794
85. qos2CbicOutlost = 314733
86. qos3CbicOutlost = 314727
87. bpduInlost = 0
88. qos0Inlost = 0
89. qos1Inlost = 0
90. qos2Inlost = 0
91. qos3Inlost = 0
92. qos4Inlost = 0
93. qos5Inlost = 0
94. qos6Inlost = 0
95. qos7Inlost = 0
96. qos8Inlost = 0
97. qos9Inlost = 0
98. qos10Inlost = 0
99. qos11Inlost = 0
100. qos12Inlost = 0
101. qos13Inlost = 0
102. qos14Inlost = 0
103. qos15Inlost = 0
104. pqueInlost = 0
105. Overruns = 0
106. maxIndex = 0
0. ifOutDiscards (ports 41-48) = 1484971158

InayathUlla Sharieff · ‎09-16-2015

Hi,

From your configuration and show queueing output you do not have QOS enabled. This means that all traffic has an equal
chance of using the buffers and will not be dropped due to higher priority traffic
arriving. Since all traffic has the full use of bandwidth and buffer space we will not
drop until either have been overran.

That leads me into the next point that these output drops are clearly coming from too much
traffic destined to this interface.

All QOS 0 traffic is being dropped because all QOS tags have been assigned to queue 1. You
can see the queue/cos map above the drop counter.

So can we try increasing the hold value on the interface?

int gx/x

(config-if)#hold-queue 4096 out

end

Is it connected via single mode or multimode lnk? If in case you are using single mode can you try with multimode?

Else the other option would be to span the port and check if the traffic is expected. make sure the capture file taken is to be checked with millisecond to see the burst of the interface.

HTH

regards

Inayath