ARP input causes High CPU utilization 3750

[AhmedElGarf]

Hi all,

I have a critical issue , my CPU reaches 99% from now and then and when i check the cpu proc I can see ARP Input process is causing that .. I dont know what the reason , can you help me ?

[Mark Malone]

Hi
are you using static routes to next hop interfaces instead of actual ips ? that drives up arp cpu pretty quick

is there anything else with the arp running high ?
did you capture the show tech or even the show proc cpu sorted as its happening

[AhmedElGarf]

The defualt route is next hop IP ,

Sometimes when I check the process I found HULC led proccess high also , but the ARP input is more than 50%!!

[Mark Malone]

The HULC is a known bug
is the input process high at the same time as the arp
could also be a malicious arp stream is there many incomplete arps in the arp table ?
if you cant post the show tech then id advise you to run it through the cli analyzer tool to see what it picks up or TAC it , there can be a number of reasons the arp is high and driving up the cpu , sounds like though there maybe something buggy in the software

what ios code are you on

[AhmedElGarf]

attahced show tech , there's no imcomplete requests now 

and here's the show ip traffic

ABB-CoreSTK# show ip traffic
IP statistics:
Rcvd: 1143608357 total, 952447409 local destination
742 format errors, 255 checksum errors, 160578516 bad hop count
8 unknown protocol, 6077545 not a gateway
0 security failures, 0 bad options, 11096387 with options
Opts: 23 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 23 record route
0 stream ID, 0 strict source route, 11096364 alert, 0 cipso, 0 ump
0 other
Frags: 596 reassembled, 188 timeouts, 0 couldn't reassemble
1752 fragmented, 0 couldn't fragment
Bcast: 591962148 received, 11473382 sent
Mcast: 24964990 received, 224709935 sent
Sent: 671207769 generated, 2589713719 forwarded
Drop: 1559 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero

ICMP statistics:
Rcvd: 1 format errors, 364 checksum errors, 1 redirects, 51200 unreachable
8439350 echo, 131726 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 5 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 622345 redirects, 1212334 unreachable, 22791900 echo, 8439350 echo reply
0 mask requests, 0 mask replies, 0 quench, 5 timestamp
0 info reply, 55517833 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 18087356 total, 0 checksum errors, 184 no port
Sent: 16682653 total

UDP statistics:
Rcvd: 900763980 total, 18 checksum errors, 85876228 no port
Sent: 351156193 total, 469892524 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

EIGRP-IPv4 statistics:
Rcvd: 17628454 total
Sent: 0 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0

OSPF statistics:
Rcvd: 7343257 total, 0 checksum errors
6420172 hello, 37 database desc, 5 link state req
505673 link state updates, 417358 link state acks

Sent: 9022087 total
8569806 hello, 47 database desc, 4 link state req
57024 link state updates, 395234 link state acks

ARP statistics:
Rcvd: 379354661 requests, 50339129 replies, 607 reverse, 0 other
Sent: 54182082 requests, 272872461 replies (20181949 proxy), 0 reverse
Drop due to input queue full: 46605163

[Mark Malone]

please post the show version too so it can be ran in the cli analyzer its missing from the tech and doesnt work without it

at the minute while that was taken there is no arp issue its like couple of percent
9 610218701 415662474 1468 1.59% 2.56% 2.39% 0 ARP Input

i see stp process is in pay can you check there are no constant stp changes which would drive up cpu
use this command to see recent changes
sh spanning-tree detail | i ieee|occurr|from|is exec


also i would add a cpu script so when it hits 70% again it will collect the info in real time and dump it in flash so you can see whats happening while its being hammered

event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val "70" exit-time 10 poll-interval 5
action 0.1 syslog msg "CPU Utilization is high"
action 0.2 cli command "enable"
action 0.4 cli command "show log | append flash:CPU_Profile.txt"
action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"
action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"
action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"
action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"
action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"
action 1.0 cli command "no event manager applet High_CPU"
action 1.1 cli command "end"

[Leo Laohoo]

Post the complete output to the following commands:
1. sh version;
2. sh controll e <PORT>