Re: 2651 FE Int. Input queue drops.

mbellears2 · ‎03-07-2005

We have a 2651, performing Dot1q, and are seeing large amounts of input queue drops:

FastEthernet0/0 is up, line protocol is up

Hardware is AmdFE, address is 0007.855b.5840 (bia 0007.855b.5840)

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 3/255, rxload 3/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 20:39:03

Input queue: 4/75/5861/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: weighted fair

Average CPU utilisation is ~15%, peak is ~45%

IP appears to be the culprit(No big suprise here):

#show processes CPU | i ^PID|Input

11 162579 697759 233 0.00% 0.00% 0.00% 0 ARP Input

26 73814 721988 102 0.00% 0.00% 0.00% 0 Net Input

31 12072602471755006842 687 20.88% 16.41% 14.13% 0 IP Input

43 0 1 0 0.00% 0.00% 0.00% 0 Probe Input

44 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

I'm not seeing any cache misses:

#show interfaces fastethernet 0/0 switching

FastEthernet0/0

Throttle count 0

Drops RP 5861 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 83523 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out

Other Process 721988 34655622 2165840 129950400

Cache misses 0

Fast 0 0 135688 204346128

Auton/SSE 0 0 0 0

IP Process 1908005005 595021258 1852686177 1263674362

Cache misses 0

Fast 2074538484 2144311152 2074538484 2145664076

Auton/SSE 0 0 0 0

ARP Process 338824 20329440 3838524 245665536

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

CDP Process 360995 145480985 360997 116241001

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

So it appears to be caused by packets that are destined for the router.

So, I tried using "show ip traffic"(Attached) to determine which higher-layer protocol is congesting the input queue, but I'm not entirely sure what I should be looking for.

Any assistance would be greatly appreciated.

Regards,

MB

allan.thomas · ‎03-07-2005

You are quite right, from the show interfaces fastethernet 0/0 switching it is apparent that there are no cache misses which suggests the packets are destined for the router.

Also you have already identified, IP is the protocol which is causing the suspected input queue congestion.

From the show ip traffic you attached, it is evident that there are a high number of enscapsulation failures, which usually indicates that the router had no ARP request entry, and consequently the datagram was dropped.

Further to this drops are also evident where there is no route in the routing table, hence the datagram is blackholed, and will be dropped. This could be a symptom of network instability, and frequent routing updates could congest the input queue.

Other sources of excessive traffic directed to the router itself could be SNMP, and ICMP. The show ip traffic indicates a high number of ICMP unreachable being sent which implies that there is no route.

Most of the time when congestion occurs one type of packet is present in large quantities, run the show buffers input-interface fastethernet 0/0 which will highlight the protocol evident at that time.

for example, at the bottom of the following output PROT 1 is shown, which is Protocol 1 ICMP.

Buffer information for Small buffer at 0x612EAF3C

data_area 0x7896E84, refcount 1, next 0x0, flags 0x0

linktype 7 (IP), enctype 0 (None), encsize 46, rxtype 0

if_input 0x6159D340 (FastEthernet3/2), if_output 0x0 (None)

inputtime 0x0, outputtime 0x0, oqnumber 65535

datagramstart 0x7896ED8, datagramsize 728, maximum size 65436

mac_start 0x7896ED8, addr_start 0x7896ED8, info_start 0x0

network_start 0x7896ED8, transport_start 0x0

source: 212.176.72.138, destination: 212.111.64.174, id: 0xAAB8,

ttl: 118, prot: 1

In summary I would be particularly be concerned with UDP and ICMP, SMNP also falls within UDP, as does ICMP. I would therefore concerntrate on debugging the received ip packets.

Use ACLs to filter traffic whilst debugging, for example, If the show buffers interface instigates ICMP, then configure ACLs to filter ICMP, this would pinpoint the source.

mbellears2 · ‎03-07-2005

Appreciate the response/assistance Allan....I "believe" I have located the culprit - I did not have ip cef enabled!!

CPU Utilisation has now decreased:

#show processes CPU | i ^PID|Input

11 162716 698291 233 0.00% 0.00% 0.00% 0 ARP Input

26 73862 722517 102 0.00% 0.00% 0.00% 0 Net Input

31 12092254911757472187 688 0.00% 0.07% 0.08% 0 IP Input

43 0 1 0 0.00% 0.00% 0.00% 0 Probe Input

44 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

And not seeing any more input drops (For the last 5min. at least!):

#show interfaces fastEthernet 0/0

FastEthernet0/0 is up, line protocol is up

Hardware is AmdFE, address is 0007.855b.5840 (bia 0007.855b.5840)

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 3/255, rxload 3/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:03:53

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: weighted fair

Output queue: 0/1000/64/0 (size/max total/threshold/drops)

Conversations 0/2/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

Available Bandwidth 75000 kilobits/sec

5 minute input rate 1224000 bits/sec, 513 packets/sec

5 minute output rate 1220000 bits/sec, 507 packets/sec

Regards,

MB