output drop issue...

farooq_0923@hotmail.com · ‎05-07-2013

Hi Guys,

I am having a issue where i see huge output drops on trunk .....basically i have a access layer switch which is conected to core , core's trunk is gig \n access layer has 100 meg....instead of seeing errors on 100meg i see it on gig interface....i have host A conected to access layer which polls servers connectwed behind another core switch (say host B) ...every now and then host A idrop packet and send false alert out for unpoll server as down....now i traced path and done ping test from HOST A to HOST B .....from Host A to get to (PING) servers it takes Max 25 ms for 30000 pings ( min is 1ms, avergae is 2msec)...for HIOSt B to get HOST A min 1msec , average 3msec and max 100msec for 1000 pings.....i even traced the path back ( mac to port tracing )and HOST hits one main core switch and then destination and HOST B hits two core switches and then HOST A....i see out put drops on GIG trunk of core...

CORE switch trunk

GigabitEthernet3/37 is up, line protocol is up (connected)

Hardware is C6k 1000Mb 802.3, address is 001e.13dd.e5b4 (bia 001e.13dd.e5b4)

Description: trunk to it area switch

MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,

reliability 255/255, txload 5/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s

input flow-control is off, output flow-control is off

Clock mode is auto

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:34, output 00:00:50, output hang never

Last clearing of "show interface" counters 1w4d

Input queue: 0/2000/22/0 (size/max/drops/flushes); Total output drops: 318966450

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 124000 bits/sec, 208 packets/sec

5 minute output rate 2335000 bits/sec, 844 packets/sec

423686111 packets input, 75694893840 bytes, 0 no buffer

Received 771518 broadcasts (707699 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 11 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

2977282761 packets output, 2042066847687 bytes, 0 underruns

2 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

SHow interface counter error

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

Gi3/37 0 0 2 0 0 318966450

SHow ip traffic on core

IP statistics:
Rcvd: 154315762 total, 982345507 local destination
         107 format errors, 1704 checksum errors, 65838464 bad hop count
         0 unknown protocol, 22423094 not a gateway
         0 security failures, 0 bad options, 26827831 with options
Opts: 41092 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 8 record route
         0 stream ID, 0 strict source route, 26786739 alert, 0 cipso, 0 ump
         0 other
Frags: 300 reassembled, 0 timeouts, 0 couldn't reassemble
         204750 fragmented, 0 couldn't fragment
Bcast: 565095059 received, 2084899 sent
Mcast: 396305142 received, 326552877 sent
Sent: 304134024 generated, 2119036473 forwarded
Drop: 272133326 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 12466 forced drop
Drop: 81 packets with source IP address zero

ICMP statistics:
Rcvd: 0 format errors, 33700 checksum errors, 2196 redirects, 346621 unreachable
        13450588 echo, 8187 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        284 irdp solicitations, 0 irdp advertisements
Sent: 23639 redirects, 11737 unreachable, 8383 echo, 13450505 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 2668767 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

PIMv2 statistics: Sent/Received
Total: 55361949/147580009, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 9646881/9648389
Join/Prunes: 0/0, Asserts: 45715068/137931620, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 4916042/30310848, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 4765648/3, Host Reports: 150394/27250088, Host Leaves: 0/1045333
DVMRP: 0/0, PIM: 0/0

UDP statistics:
Rcvd: 789512758 total, 81 checksum errors, 68072949 no port
Sent: 226593422 total, 1337189545 forwarded broadcasts

TCP statistics:
Rcvd: 1100399 total, 0 checksum errors, 2336 no port
Sent: 1099626 total

OSPF statistics:
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

Sent: 0 total

EGP statistics:
Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener
Sent: 0 total

Probe statistics:
Rcvd: 0 address requests, 0 address replies
0 proxy name requests, 0 where-is requests, 0 other
Sent: 0 address requests, 0 address replies (0 proxy)
0 proxy name replies, 0 where-is replies

IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total

ARP statistics:
Rcvd: 476570723 requests, 31382270 replies, 40485 reverse, 0 other
Sent: 57319280 requests, 80166745 replies (79900818 proxy), 0 reverse
Drop due to input queue full: 179517

Access layer switch ip traffci

IP statistics:
Rcvd: 20444807 total, 9491341 local destination
         0 format errors, 0 checksum errors, 10526295 bad hop count
         0 unknown protocol, 4422 not a gateway
         0 security failures, 0 bad options, 6782252 with options
Opts: 40968 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 6741284 alert, 0 cipso, 0 ump
         0 other
Frags: 5998 reassembled, 1 timeouts, 0 couldn't reassemble
         0 fragmented, 0 couldn't fragment
Bcast: 1211668 received, 19 sent
Mcast: 0 received, 0 sent
Sent: 7814147 generated, 2 forwarded
Drop: 153 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop
         0 options denied, 0 source IP address zero

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 18470 unreachable
        7071061 echo, 128440 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 8 unreachable, 128495 echo, 7071061 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 1 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 442950 total, 0 checksum errors, 844 no port
Sent: 522227 total

UDP statistics:
Rcvd: 1830400 total, 0 checksum errors, 1202334 no port
Sent: 92514 total, 2976 forwarded broadcasts

IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total

ARP statistics:
Rcvd: 12419723 requests, 13914426 replies, 0 reverse, 0 other
Sent: 3113 requests, 2782 replies (16 proxy), 0 reverse
Drop due to input queue full: 0

FastEthernet1/0/1 is up, line protocol is up (connected)

Hardware is Fast Ethernet, address is 001f.9d4f.e003 (bia 001f.9d4f.e003)

Description: floorport main

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 6/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, media type is 10/100BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 1w5d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 2674000 bits/sec, 897 packets/sec

5 minute output rate 128000 bits/sec, 215 packets/sec

3096311877 packets input, 2122061396037 bytes, 0 no buffer

Received 324515796 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 248145891 multicast, 0 pause input

0 input packets with dribble condition detected

Can any one advise please ....iam think of upgrading from 100 meg to gig but i still should see these errors on 100 trunk not core....and also all the cores got 10gig connection between each other .....

paul driver · ‎05-07-2013

Hello
Output drops usually indicate congestion onI the link. however from the readouts you have posted don't show any high utilisation on the interfaces.

I do see you have default mtu of 1500 - possibility this needs to higher to support the multiple vlan tags of the dot1q trunks

Do you have any Qos applied on these switches?

Res
Paul

Sent from Cisco Technical Support iPad App

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

farooq_0923@hotmail.com · ‎05-07-2013

hi paul,

thanks for your reply, yes auto qos has been configured ....as far as utilization is concerned access layer switch has high voulme of traffic during peak hours, mind out put is from ooh.... but y dnt i see any drops on 100 meg trunk? ....

Leo Laohoo · ‎05-07-2013

Ok, output drops means that your interface is pushing as much data as it can but the remote end device is sending back a PAUSE frame.

So the data that the switch cannot deliver goes into a bucket or buffer. This buffer gets full and this is why you are seeing output drops.

Now, the question is WHERE is the fault.

One easy test is to bring down your interface speed to 10 Mbps and see if the output drops stops incrementing. If it does, then I'd recommend you look at the remote end.

farooq_0923@hotmail.com · ‎05-07-2013

thanks leo,

my host A is trying to talk to host B.... host A is connected to access layer switch which connects to core A ... host A DG points to hsrp ip for vlan configured on core A... core a send all traffic to coreB where host B is ... so if i see output drops on core A trunk ( trunk between host A access layer switch and cor A) this means core A sends but all traffic to Core B but due to congestion Core B cant take all frames and core B trunk puts it in buffern then drop... am i right? ... all the three core's have trunks configured two cores areusing hsrp third is datacenter vss ...lately more 48 port gig switches have been added to network n uplinks arenot increased they are only 10 gigs....